Automatic Sample-by-sample Model Selection Between Two Off-the-shelf Classifiers

If one could predict which of two classifiers will correctly classify a particular sample, then one could use the better classifier. Continuing this selection process throughout the data set should result in improved accuracy over either classifier alone. Fortunately, scalar measures which relate to the degree of confidence that we have in a classification can be computed for most common classifiers (Hastie & Tibshirani 1996). Some examples of confidence measures are distance from a linear discriminant separating plane (Duda & Hart 1973), distance to the nearest neighbor, distance to the nearest unlike neighbor, and distance to the center of correctly classified training data. We propose to apply discriminant analysis to the confidence measures, producing a rule which determines when one classifier is expected to be more accurate than the other. Let q1(x) and q2(x) be scalar functions for the confidence measures of two off-the-shelf classifiers. Each sample, xi, is mapped to (q1(xi), q2(xi)) in the decision space for selecting a classifier, thus the decision space has only two dimensions. Observe that the sample space has d-dimensions where d is the number of features in the sample. In this respect the dimensionality of selecting the classifier is reduced from d to 2. In order to select the better classifier, we need an estimate of where each classifier succeeds or fails. Both classifiers are applied to each training sample to create this estimate. Classifiers which never misclassify a training sample, such as nearest neighbors, are evaluated by leave-one-out runs. Each training sample now has two confidence values, one from each confidence function. It is also known whether each classifier has correctly classified each of the training samples. This classification information is used to associate a value selected from {−1, 0, 1} with each training sample. This value is termed correctness.